feat: add Lambda SnapStart support - #831
Conversation
Switch from the git-branch dependency to the published lambda_http/lambda_runtime 1.3.0 from crates.io, removing the release blocker. Also fix the integration test body-reader helpers to accept the BoxBody response type, and resolve a clippy ok().expect() lint.
hook_target returned Ok(None) for a configured path that canonicalizes to the
root ("/", "//", "/..", "/.", "/foo/..", "/%2f"), silently disabling the guard.
But after_restore POSTs the RAW configured path — it reads after_restore_path,
not the guard target — so the hook still fired at "/". The two diverged with no
diagnostic: with AWS_LWA_SNAPSTART_AFTER_RESTORE_PATH=/.. the adapter POSTs to
"/" on every restore, which is a 405 on both FastAPI examples (they declare only
`@app.get("/")`), and post_hook treats any non-2xx as fatal — so every restore
failed and nothing explained why.
Reject it instead. The guard cannot cover the root without returning 403 for all
normal traffic, and the docs require a hook path "your normal application traffic
does not use", which the root never is. Same rule as the % cases: if the adapter
cannot guard the route, it refuses to run with it rather than starting up with a
state-mutating route reachable or a hook that fails every restore.
Unset and empty still mean "no hook" and are unaffected.
…r SnapStart build_client applied pool_max_idle_per_host(0) whenever AWS_LAMBDA_INITIALIZATION_TYPE=snap-start, and that variable stays set for the whole lifetime of a restored environment. Both call sites went through it, so the client rebuilt in after_restore -- the one Adapter::client() returns for every invocation after a restore -- also never retained a connection. The configured idle keep-alive was therefore a no-op on exactly the functions this feature targets, and every invocation opened a fresh TCP connection to the inner app for the life of the environment, consuming a file descriptor each time against Lambda's limit. The snapshot hazard only applies to the client built BEFORE the snapshot. A client built inside after_restore starts with an empty pool and cannot hold a snapshotted connection, so it is safe for it to pool normally. build_client no longer reads the environment; the caller decides, so the post-restore rebuild cannot silently inherit the pre-snapshot restriction. Adapter::new passes Duration::ZERO under SnapStart via the new base_client_idle_timeout, which disables idle keep-alive for the pre-snapshot client -- measured equivalent to pool_max_idle_per_host(0), including for back-to-back requests. The configured value is retained on Adapter::pool_idle_timeout and used for the after-restore rebuild. This keeps the pre-snapshot client safe by construction, so a consumer driving the Service impl directly (who never triggers the after-restore hook) is still protected against hyper#3810, and it removes the post-restore path's dependence on AWS_LAMBDA_INITIALIZATION_TYPE. Also makes the SnapStartHooks::pool_idle_timeout field comment true: the post-restore client now really does honor the configured value.
…ness branch
Findings from a final systematic pass over the branch.
1. hook_target short-circuits on configured.is_empty() and returns Ok(None) ("no
hook"), but Adapter::new stored the raw Some(""), which run() hands to
SnapStartHooks. before_snapshot/after_restore then took their `if let
Some(path)` branch and called post_hook(.., ""), and Url::set_path("") yields
"/" -- so the adapter POSTed to the unguarded application root on every
lifecycle event (405 on both FastAPI examples, which post_hook treats as
fatal). This is the same guard-versus-POST divergence the root-collapse
rejection closed; "" slipped past by returning before canonicalization.
Adapter::new now normalizes an empty hook path to None before anything reads
it, so both sides agree by construction and the documented "empty means unset"
semantics are preserved. Only reachable via a directly constructed
AdapterOptions -- env-derived options already drop empties.
2. readiness::wait_until_ready drives Retry::spawn over an unbounded
FixedInterval, so it can only return ready or never return. Its bool, and the
`if !ready` branches plus "readiness check failed" errors in
check_readiness_with_timeout / check_readiness_unbounded, were unreachable.
Removed; wait_until_ready now returns (). check_init_health's ready_at_init
comes from whether the wait COMPLETED within its bound, which is what the
value always meant. Documented that an unbounded post-restore wait holds the
restore open until Lambda's own timeout, with the escalating "app is not ready
after {}ms" log as the adapter-side signal, and that
AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS converts that into /restore/error.
Doc corrections, no behavior change:
- canonicalize_hook_path's rustdoc listed a control/null byte as a None case, but
control bytes are stripped and canonicalization continues -- they WIDEN the
blocked class. Stale in the fail-open direction. The same paragraph still
described the HookTarget::Raw fallback deleted in ca56fd9, and the guard
comment in fetch_response repeated the control-byte claim.
- build_client / base_client_idle_timeout claimed Duration::ZERO means no
connection is "retained". It does not: hyper's pool stays enabled
(Config::is_enabled is max_idle_per_host > 0), so the socket is still parked and
captured in the snapshot; ZERO guarantees it is evicted on checkout rather than
reused. Documented the real guarantee, and the init-time reconnect cost it
carries (27 connections per 300ms of readiness polling versus 1) -- confined to
init, and unchanged from the pool_max_idle_per_host(0) behavior it replaced.
- SnapStartHooks::client is used for the before-checkpoint hook only;
after_restore deliberately uses the fresh client.
- The guide's rejection rule said "a path containing a percent sign", stricter
than the code, which rejects only when the DECODED form contains one --
/snapstart/%61fter is accepted and guarded as /snapstart/after.
- Two public-docs-link-to-private-item rustdoc warnings; cargo doc --no-deps is
now clean.
- Unused `import os` in the zip example.
…g it be50614 replaced pool_max_idle_per_host(0) with pool_idle_timeout(Duration::ZERO) for the pre-snapshot client, and claimed the two were "identical on reuse". They are not, in exactly the scenario the original workaround was written for. A zero idle timeout leaves hyper's pool ENABLED (Config::is_enabled() is max_idle_per_host > 0), so the connection is parked in the idle map and reuse is decided at checkout by `now.saturating_duration_since(idle_at) > timeout`. That saturates to ZERO when the recorded instant is ahead of `now`, and ZERO > ZERO is false -- so the entry counts as fresh and is handed out. A monotonic clock that has not advanced across a restore is precisely the condition hyper#3810 / rust-lang/rust#79462 describe, so the guarantee rested on the very clock the workaround exists to distrust. is_closed() does not catch it either: the app process was restored from the same snapshot and never sent a FIN. Under run() this is masked, because after_restore publishes a fresh client before any invocation. The only exposed path is a consumer driving the Service impl directly -- which is the sole reason the pre-snapshot restriction exists, so the protection was vacuous for its only beneficiary. build_client now takes an explicit Pooling parameter: Disabled sets pool_max_idle_per_host(0) (pool off, no clock consulted) and Adapter::new uses it under SnapStart via base_client_pooling; the after-restore rebuild passes Enabled and keeps the configured AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS, so everything gained in be50614 on the invocation path is retained. test_adapter_new_client_never_pools_under_snapstart could not catch this: it sleeps 40ms between requests, so elapsed() is non-zero and it passes either way. The new test_pre_snapshot_client_pool_is_disabled_not_merely_expiring observes the connection's lifetime instead -- whether the socket is dropped or parked after one request -- which no clock reading can satisfy.
2033c84 to
81fadbd
Compare
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..81fadbd
Files: 27
Comments: 3
Most items from the prior review rounds are now resolved in the code I read: check_init_health returns Err on sync-init timeout (src/lib.rs, and src/main.rs propagates it), Duration::try_from_secs_f64 replaces the panicking constructor, hook_target rejects root-collapsing and %-bearing configured paths at init, both sides of the guard now normalize through Url::set_path, control bytes are stripped-and-blocked rather than passed through, pool_max_idle_per_host(0) is restored for the pre-snapshot client via base_client_pooling(), and the duplicated guard comment block is gone. The remaining findings are narrow.
Comments on lines outside the diff:
[src/lib.rs:1446] [GENERAL] hook_target fails initialization for hook paths it cannot guard (root-collapsing, literal %), but it does not detect a hook path that collides with pass_through_path. Because the pass-through rewrite runs before the guard:
if matches!(request_context, RequestContext::PassThrough) && parts.method == Method::POST {
path = self.pass_through_path.as_str();
}
// ... guard runs on this rewritten pathsetting AWS_LWA_SNAPSTART_AFTER_RESTORE_PATH=/events (the default pass-through path) makes every non-HTTP trigger event get rewritten onto the guarded route and answered with 403 instead of being delivered to the app — silently, with only a per-invocation warn!. Given the init-time validation already exists for the other unguardable cases, rejecting a hook path equal to pass_through_path there would be consistent and cheap.
…h; warn on bad env values Three findings from the latest bot pass. 1. A hook path equal to AWS_LWA_PASS_THROUGH_PATH was accepted, but the pass-through rewrite in fetch_response replaces `path` with pass_through_path for a PassThrough POST BEFORE the guard runs. So configuring the hook at /events -- the default pass-through path -- made every non-HTTP trigger event canonicalize onto the guarded route and get a 403 instead of reaching the app, silently, with only a per-invocation warn!. Adapter::new now rejects a hook path that resolves to the same route as the pass-through path, alongside the existing unguardable cases, and compares canonical routes so /Events, /events/, /./events and /%65vents are caught too. 2. duration_secs_from_env silently fell back to the default on any unparseable value -- the exact failure mode readiness_check_timeout_from_env was written to avoid. The two sibling variables also accepted different numeric formats: AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS took fractional seconds while AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS took only whole ones, so `0.5` and `4.5` parse-failed into 4s with no diagnostic. It now accepts fractional seconds like its sibling and warns on any set-but-unusable value. 3. The request path was canonicalized twice per invocation whenever both hook paths are configured -- the configuration both examples and the guide use -- because each matches_hook_path call canonicalized independently. New matches_any_hook_path canonicalizes once and compares against both targets; semantics are identical, and the no-hooks case still short-circuits before canonicalizing. matches_hook_path becomes the #[cfg(test)] single-target form. The guide's rejection list gains the pass-through case, and Adapter::new's # Errors section now names all four unguardable hook-path conditions.
Regression from de0ea31. The pass-through collision check ran let pass_through_target = hook_target(&domain, &Some(pass_through_path))?; unconditionally, so a pass_through_path that hook_target rejects aborted Adapter::new -- with a SnapStart-flavored error -- even when no hook was configured and there was therefore no guard and nothing to collide with. AWS_LWA_PASS_THROUGH_PATH is read straight from the environment with no prior validation, and "/" is a plausible setting for an app that handles non-HTTP event payloads at its root; it collapses to the root and so was rejected. The same applied to a pass-through path whose decoded form contains a literal % or a malformed escape. Existing tests missed it because they all use the default /events, which canonicalizes cleanly. The check now runs only when a hook target exists, and treats an unguardable pass-through path as "no collision" rather than an error. That is sound, not just lenient: hook targets are canonicalizable and non-empty by construction, so a request rewritten onto a root-collapsing, %-bearing, or non-canonicalizable pass-through path can never canonicalize onto one -- the guard would pass it through anyway.
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..f6679db
Files: 27 (see note on src/lib.rs)
Comments: 4
│ Note: .lambda-tooling-pr-reviewer/diff/file_src_lib.rs.json contains "(no patch available; status=modified)", and the workspace copy of src/lib.rs is the pre-PR state (no snapstart, build_client, canonicalize_hook_path, or restored_client symbols in it). The largest change in this PR (+1689/-94) therefore could not be read. Findings 1–3 below are prior-round issues on that file that the author never explicitly dismissed; they are re-raised unverified, and each notes what would resolve it.
`let _ = self.restored_client.set(fresh.clone())` discarded the "already set" case and then used `fresh` for steps 2 and 3. If the cell were already populated, the hook POST and the readiness check would run over a client no request can reach, so the restore would report healthy on the basis of something the request path never touches -- with no signal anywhere. New publish_or_adopt returns whichever client invocations will actually use: `fresh` when it wins the race, otherwise the already-published one. That removes the divergence rather than merely reporting it, and warns so an unexpected second lifecycle run is visible in logs. Latent today, since lambda_runtime drives the restore lifecycle once. The test pins the property directly via Arc::ptr_eq -- a second call must return the first client, not its own -- so it cannot become real.
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..928bc8c
Files: 26 of 27 (see note)
Comments: 4
Note on coverage: .lambda-tooling-pr-reviewer/diff/file_src_lib.rs.json contains "patch": "(no patch available; status=modified)", so the largest change in this PR (src/lib.rs, +1689/-94 — the hook guard, env parsing, build_client, register_and_run, fetch_response) could not be read. The workspace copy of src/lib.rs is the pre-PR state (no snapstart, Pooling, build_client, or canonicalize_hook_path symbols present), so it is not a substitute. Comments 1–3 are therefore re-raises of prior findings on that file that carry no author response; they are based on the code quoted in the earlier threads, not on the current head, and should be re-verified against it. Everything else (src/snapstart.rs, src/readiness.rs, src/main.rs, tests/integ_tests/main.rs, both examples, docs, Cargo.toml/Cargo.lock, CHANGELOG.md) was reviewed in full and I found no further issues.
register_extension_internal built its RAPID client with default pooling, so a connection to the Runtime API could be parked in the idle pool and captured in a SnapStart snapshot. That is the same hazard lambda_runtime handles by calling reset_pool() on its own RAPID client during the restore lifecycle -- but nothing resets or re-establishes this one, and register_default_extension terminates the process with exit(1) when its request fails, so handing out a dead connection after restore would kill the restored environment before it serves anything. Pooling costs nothing to give up here: the client issues exactly two requests, `register` and then the long poll for the first extension event, and the long poll's in-flight connection is unaffected by the idle-pool setting. Disabled unconditionally rather than gated on AWS_LAMBDA_INITIALIZATION_TYPE, since there is no case where reuse helps. Extracted as runtime_api_client so the property is testable; the test observes connection lifetime (dropped vs parked), and I confirmed it fails against a default-pooled client before the change.
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..db37e17
Files: 27 (16 reviewed in detail; lock file, .gitignores, and event fixtures skipped)
Comments: 1
A caveat before the findings: .lambda-tooling-pr-reviewer/diff/file_src_lib.rs.json contains "patch": "(no patch available; status=modified)", so the diff for src/lib.rs (+1729/−95) was not available, and the working-tree copy is the pre-PR state. That file holds the hook guard, canonicalize_hook_path, the env-var parsing for the two new timeouts, and the reworked Adapter::new/fetch_response. I could not verify or refute the unresolved prior findings scoped to it (the Duration::from_secs_f64 overflow path, the configured-vs-request canonicalization asymmetry, the root/pass-through collapse guards, AdapterOptions not being #[non_exhaustive]). Several of those look addressed judging by src/main.rs, the integ-test changes, and the new docs — the cold-start timeout now propagates (check_init_health().await?) and the guide documents startup rejection of root-collapsing, percent-ambiguous, and pass-through-colliding hook paths — but that is inference from adjacent files, not verification. Re-run the review with the src/lib.rs patch present before treating it as reviewed.
The pre-PR comment explaining why pooling was disabled under SnapStart was the only record of the reason, and it was deleted. Restore it with the measurement that settles it, taken from a SnapStart container function deployed from this branch: across the restore: monotonic +0.54s while wall +161s after the restore: monotonic +6.079s / +6.059s vs wall +6.1s / +6.0s CLOCK_MONOTONIC does not advance across the snapshot gap, but never goes backwards, and after the restore it tracks wall time exactly. So the anomaly is confined to the boundary, which is what makes the two sites correct in opposite directions: - Adapter::new (pre-snapshot) must have the pool OFF. hyper decides reuse with `now.saturating_duration_since(idle_at) > idle_timeout`, so an entry pooled before the snapshot reads as ~0.5s idle after restore however long the snapshot sat -- fresh, and dead. No idle timeout fixes that, including Duration::ZERO, since ZERO > ZERO is false. This is hyper#3810 / rust-lang/rust#79462. - after_restore may have the pool ON. Every entry it holds is post-boundary, where accounting is reliable; verified live with idle gaps longer than the configured 4s keep-alive all succeeding. This is also the only way AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS affects the invocations that serve traffic. Comment-only; no behavior change.
The guide and both example READMEs warn that the 403 hook guard only exists while the adapter is in the request path; the top-level README presented the guard without it, and it is the most widely read of the four. Its own opening advertises that the same image runs on EC2, Fargate and local machines -- exactly the deployments where the hook routes are reachable and unauthenticated. All four docs now carry the caveat.
Findings from the final systematic pass. 1. before_snapshot was the only path into the application that was not readiness-gated. With AWS_LWA_ASYNC_INIT=true, check_init_health gives up at 9.8s and returns Ok(()) with ready_at_init=false so the app can keep booting; run() then drives snapstart_lifecycle straight into before_snapshot, which POSTed immediately. For an app that has not bound its port the POST fails at once with ECONNREFUSED -- the 60s HOOK_TIMEOUT never applies to a refusal -- and lambda_runtime reports it to /init/error, so publishing the SnapStart version fails with what looks like an application bug. That is exactly the slow-booting app async_init exists for. Both hooks now go through ensure_ready, bounded by AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS when set. The timeout error names the phase, so an initialization failure is distinguishable from a restore failure. check_readiness_with_timeout / check_readiness_unbounded collapse into ensure_ready / ensure_ready_with_timeout. 2. The five pooling tests mutated the process-global AWS_LAMBDA_INITIALIZATION_TYPE with directly conflicting expectations, so under plain `cargo test` -- the runner a contributor following the standard Rust workflow uses, where nextest's per-test process isolation does not apply -- they could interleave and invert each other's assertions. Those are the assertions guarding the snapshot-connection hazard, so a silent inversion is the worst case. Four of them now pass Pooling explicitly to build_client and touch no environment at all. The env-var side collapses into one test, test_base_client_pooling_from_env, per the convention the neighbouring env-mutating tests already follow; verified it fails when the decision is inverted. 3. AdapterOptions::pool_idle_timeout's rustdoc and the CHANGELOG said AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS takes "whole seconds", left stale by the commit that made it accept fractional seconds and pinned 0.5 -> 500ms in a test. 4. CHANGELOG now records the lambda_http 1.1.1 -> 1.3.0 bump and that it changes the Cookie header on every deployment: a multi-entry API Gateway v2 cookies array is joined with "; " instead of ";", so apps see `a=1; b=2`. RFC 6265 form and universally accepted, but it is a plain-request-path change that was undocumented.
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..85ad8cb
Files: 27
Comments: 3
Comments on lines outside the diff:
[src/lib.rs:0] [SECURITY] Re-raising (previously flagged, no author response): the hook-path guard's equivalence class does not include path parameters (matrix variables). The guard canonicalizes percent-encoding, collapses // and ./.., and compares case-insensitively — the guide enumerates exactly these — but a ;-suffixed segment is kept verbatim by the url crate, so:
request: POST /snapstart/after;x=1
set_path: /snapstart/after;x=1 (';' left in the path)
guard: != /snapstart/after -> forwarded
Jakarta Servlet containers and Spring strip ;-parameters from each segment before routing, so a request the guard forwards can still land on the hook handler. That is a reachable, unauthenticated, state-mutating route on the frameworks this repo supports (examples/springboot, examples/springboot-zip, examples/javalin-zip). Suggest truncating each segment at the first ; inside canonicalize_hook_path so both sides land in the same equivalence class.
[src/lib.rs:0] [SECURITY] Re-raising (previously flagged, no author response): the guard fails open on the request side, which contradicts its own documented contract. canonicalize_hook_path's rustdoc states:
/// Returns None only for genuinely undecidable inputs (a malformed % escape,
/// non-UTF-8 after decoding, or a control/null byte). The caller treats None
/// as "reject" (fail closed).
but on the request side None is turned into "not the hook" and the request is forwarded. Configured-side None is rejected at startup (that part is fail-closed and correct); the request side is the half that matters for the guard, and there the undecidable branch admits the request instead of returning 403. Even if no concrete bypass spelling exists for a given framework today, a security guard whose comment says "fail closed" while the code fails open is very likely to be broken by a future edit. Either return 403 on None for request paths, or correct the doc comment to state that the request side deliberately fails open and why.
[src/lib.rs:0] [BUG] Re-raising (previously flagged, no author response): AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS (and the same pattern for AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS) can panic at startup on an out-of-range value. As quoted in the earlier thread:
.filter(|secs| secs.is_finite() && secs >= 0.0)
.map(Duration::from_secs_f64)Duration::from_secs_f64 panics on overflow, not just on NaN/infinity/negatives. AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS=1e20 parses as a finite positive f64 above Duration's max (~1.8e19 s), so it passes the filter and panics — the adapter aborts during init on a mistyped env var rather than warning and falling back, which is the behavior the README and guide both promise ("a set-but-<= 0 or malformed value is ignored with a warn!"). Use Duration::try_from_secs_f64 (or bound the value) and route the error into the existing fallback-with-warn! path. Note: I could not read the current src/lib.rs to confirm this line survives at HEAD — if the parser was already switched to try_from_secs_f64, this is resolved and can be dismissed.
…cation The guard's summary line and a test-section banner called it "strict, fail-closed" while the detailed doc a dozen lines below correctly explains that the REQUEST side forwards an undecidable path rather than 403-ing it. Only the configured side fails closed, and the review bot has now twice read the summary, concluded the request side returns 403 on doubt, and re-raised it as a contradiction -- quoting doc text that was already replaced. The risk is not the confusion itself but the obvious "fix" it invites: turning the request-side pass-through into a 403 would reinstate the /reports/100%25 false-403 regression that pass-through exists to prevent. Both labels now state which half does which, the summary points at the reasoning rather than summarizing it away, and the term is gone from the codebase entirely (the remaining historical use in a test comment is reworded to "undecidable branch"). Comment-only; no behavior change.
The extension registers with `{"events": []}` and then parks on
GET /2020-01-01/extension/event/next forever. Nothing said why, so the standing
question "the long poll is not re-established after a SnapStart restore -- is that
a leak or a crash risk?" kept coming back.
It is neither, and the empty events list is the reason. Being registered at all is
the entire point: Lambda only delivers SIGTERM at environment shutdown when an
extension is registered, which is what the graceful-shutdown feature relies on.
Subscribing to nothing means RAPID has no event to deliver, so the long poll is
never going to resolve -- parking on it IS what keeps the extension alive.
Under SnapStart that request is captured mid-flight in the snapshot and never
re-established, which is harmless for the same reason, and RAPID's registration
state is part of the snapshotted microVM so the effect survives even though the
connection does not.
Verified on a deployed SnapStart container function: the restored environment
received SIGTERM 457s after restore, in the same log stream that logged
after_restore, and the failure path that would exit(1) never fired across ~8
minutes and five restores. A non-SnapStart control on $LATEST behaved identically
apart from the clocks (monotonic_delta == wall_delta on a normal cold start, versus
0.38s against 184.3s across a restore).
Re-registering after restore would be robustness, not a fix.
Comment-only; no behavior change.
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..db2eaba
Files: 27
Comments: 3
Comments on lines outside the diff:
[.lambda-tooling-pr-reviewer/diff/file_src_lib.rs.json:1] [GENERAL] The diff payload for src/lib.rs — by far the largest change in this PR (+1787/-95, including the SnapStart hook guard, the env-var parsing for the two new timeout variables, build_client/register_and_run, and the AWS_LWA_REMOVE_BASE_PATH semantics change) — is empty:
{"filename": "src/lib.rs", "status": "modified", "additions": 1787, "deletions": 95,
"patch": "(no patch available; status=modified)"}The workspace checkout is the pre-PR state (no src/snapstart.rs, check_init_health still returns ()), so the post-change content of that file is not available anywhere in the workspace. The two findings below are therefore based on code quoted verbatim in the earlier review threads rather than on the current file, and coverage of src/lib.rs in this pass is incomplete — please re-run the diff generation for it before treating this review as complete.
2. [BUG] src/lib.rs — readiness_check_timeout_from_env
Re-raising (previously flagged, no author response): the parser rejects <= 0 and unparseable values but not finite values outside Duration's range, and Duration::from_secs_f64 panics on overflow, not just on NaN/infinity/negatives.
match trimmed.parse::<f64>() {
Ok(secs) if secs <= 0.0 => None, // intentional unbounded, no warning
...AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS=1e20 parses as a finite positive f64 above Duration::MAX (~1.8e19 s), so the conversion panics during AdapterOptions construction and the function fails to initialize with a panic rather than the documented warn!-and-ignore path. The new docs in this PR only promise handling for <= 0 and malformed input:
a set-but-<= 0 or malformed value is ignored with a warn!
Use Duration::try_from_secs_f64 (or bound the value before converting) and route the out-of-range case into the same warn!-and-fall-back branch as a malformed value. The same applies to AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS if it shares the conversion.
3. [SECURITY] src/lib.rs — canonicalize_hook_path / matches_hook_path
Re-raising (previously flagged, no author response on this specific point — the author's replies addressed the separate normalization-divergence thread, not the fail-open direction): the guard's documented contract and its behavior on the request side point in opposite directions.
/// Returns None only for genuinely undecidable inputs (a malformed % escape,
/// non-UTF-8 after decoding, or a control/null byte). The caller treats None
/// as "reject" (fail closed).
but on the request side None is turned into "this is not the hook path" and the request is forwarded to the app. For configured values the PR does fail closed (startup rejects them, per docs/guide/src/features/snapstart.md), which makes the asymmetry easy to miss in future edits: a request path whose decoded form contains a control byte, or that carries a spelling the canonicalizer deliberately does not model (e.g. a matrix/path-parameter suffix such as /snapstart/after;x=1, which Url::set_path leaves intact but several frameworks strip before routing), skips the 403 and reaches a state-mutating route.
Two options: either make the request side match the doc — treat None as "reject" and return 403 — or, if forwarding undecidable paths is deliberate, correct the rustdoc so it states that only the configured side fails closed, and say why forwarding is safe. As written, the comment asserts a property the code does not have on the path that faces untrusted input.
Not re-raised, as the author explicitly dismissed them: the unconditional post-restore readiness check, the hook-path normalization-divergence thread, publish = false, PackageType: Image + SnapStart in the container example, and the unreleased 1.1.0 image / layer 30 pins in both examples. Several other earlier findings (the 0-means-unbounded and cold-start-failure documentation, the AWS_LWA_REMOVE_BASE_PATH doc update, the guide's missing "adapter not in the request path" warning, the contradictory example README wording, and the Adapter::new SnapStart pooling branch) appear addressed in the current diffs.
HOOK_TIMEOUT capped each hook POST at 60 seconds. It was not useful, and it could cause the failure it was meant to prevent. Lambda already bounds both phases -- the init budget for the before-checkpoint hook, the function timeout for the after-restore hook -- so there was no unbounded hang left for the adapter to guard. What the cap actually did was land wrong in both directions: - For the after-restore hook it was unreachable. The restore phase is bounded by the function timeout, which is 10s in both shipped examples and 3s by default, so Lambda always won the race and the operator never saw the adapter's message. Four docs nonetheless promised "taking longer than 60 seconds to respond fails the SnapStart phase". - For the before-checkpoint hook it was reachable but harmful: a legitimate slow drain that Lambda's much larger init budget would have allowed got killed at 60s, failing the version publish. A cap derived from the function timeout would be the defensible version, but Lambda does not expose the timeout as an environment variable and there is no invocation context during these phases to read a deadline from. The doc comment records that so nobody goes looking. post_hook_with_timeout collapses into post_hook, its timeout-injection test goes away with the behavior, and the four doc claims are corrected to say the adapter imposes no deadline and that the after-restore hook must fit inside the function timeout. Non-2xx and transport failures still fail the phase, still covered by before_snapshot_non_2xx_is_error and after_restore_publishes_client_even_when_hook_fails.
There was a problem hiding this comment.
Code Review Results
Reviewed: 986113f..e43f76c
Files: 27
Comments: 4
│ Note on diff coverage: .lambda-tooling-pr-reviewer/diff/file_src_lib.rs.json contains "patch": "(no patch available; status=modified)", and the workspace copy of src/lib.rs is the pre-PR state (no snapstart, build_client, or canonicalize_hook_path symbols). The largest part of this PR (+1787/-95) was therefore not reviewable. Comments 2–4 re-raise prior unresolved findings against that file; I could not confirm whether the current revision addresses them. Comment 1 and the SnapStart/readiness/examples analysis below are verified against the available patches.
| - Add SnapStart support. The adapter notifies your web application at the SnapStart | ||
| boundary via two opt-in HTTP hooks — `AWS_LWA_SNAPSTART_BEFORE_CHECKPOINT_PATH` | ||
| (before checkpoint) and `AWS_LWA_SNAPSTART_AFTER_RESTORE_PATH` (after restore) — | ||
| so it can drain and re-establish connections. Each hook call is bounded by a |
There was a problem hiding this comment.
[GENERAL] The changelog states the hook calls are bounded, but the shipped code deliberately removed that bound and the other docs say the opposite.
so it can drain and re-establish connections. Each hook call is bounded by a
60-second timeout.
SnapStartHooks::post_hook in src/snapstart.rs has no timeout, and its own doc comment says so explicitly: "Deliberately unbounded... This previously capped the wait at 60s". README.md ("The adapter does not impose its own deadline on a hook") and docs/guide/src/features/snapstart.md agree with the code. Only the changelog entry is stale — it will tell operators a hung after-restore hook is capped at 60s when it is actually bounded only by the function timeout (10s in both shipped examples). Drop the sentence or replace it with the "no adapter-side deadline; bounded by the function timeout" wording used elsewhere.
2. [BUG] src/lib.rs — fn readiness_check_timeout_from_env (line number unavailable, see note)
Re-raising: previously flagged with no author response, and I cannot verify it was fixed.
Duration::from_secs_f64 panics on overflow, not just on NaN/infinity/negatives. Duration's maximum is roughly 1.8e19 seconds, so a finite positive f64 above that (AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS=1e20, or a long digit string) passes an is_finite() && >= 0.0 filter and then panics. Because this is parsed during adapter construction, the failure mode is a panicking extension at init rather than the "malformed value is ignored with a warn!" behavior the new docs promise (docs/guide/src/configuration/environment-variables.md:25). Duration::try_from_secs_f64 (or an explicit upper-bound check falling back to unbounded plus a warn!) makes the code match the documented contract. The same concern applies to AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS if it shares the parsing helper.
3. [SECURITY] src/lib.rs — fns canonicalize_hook_path / matches_hook_path (line number unavailable, see note)
Re-raising: previously flagged with no author response. The author's replies in this thread addressed a different claim (that the two sides of the comparison are normalized differently), not the fail-open direction.
canonicalize_hook_path's documented contract is that None means "reject (fail closed)", but on the request side matches_hook_path reportedly turns None into "not a hook path" and forwards the request. Any request path the canonicalizer declares undecidable (malformed % escape, non-UTF-8 after decoding, control/NUL byte) then reaches the inner application instead of getting the 403. These hook routes mutate state — both new examples close and re-establish the connection pool in them — so a fail-open branch on a state-mutating control-plane route is worth closing rather than documenting the opposite. If forwarding is genuinely the intent for undecidable paths, the doc comment on canonicalize_hook_path should say so, because as written it invites a future edit to rely on a guarantee the code does not provide.
4. [SECURITY] src/lib.rs — fn canonicalize_hook_path (line number unavailable, see note)
Re-raising: previously flagged with no author response.
The canonicalizer covers percent-encoding, //, ./.., and case, but not path parameters (matrix variables): /snapstart/after;x=1 is kept verbatim and so falls outside the equivalence class the guard blocks. Servlet-based stacks (Tomcat/Spring, used by the springboot* examples) strip ;-parameters before routing, so that spelling reaches the hook handler while the guard sees a non-matching path. Starlette/FastAPI — what the two new examples use — does not strip them, so this is not exploitable in the shipped examples, but it is reachable for other supported frameworks. Stripping everything from the first ; in each segment before comparing would close it; alternatively, document that the guard does not cover matrix-parameter spellings so operators of servlet stacks know to protect the route themselves.
Nothing to flag in src/snapstart.rs, src/readiness.rs, src/main.rs, tests/integ_tests/main.rs, Cargo.toml, or the two examples. Specifically checked: publish_or_adopt correctly adopts the already-published client rather than validating an unreachable one; before_snapshot gates the hook POST on readiness (the async_init + ECONNREFUSED path is covered by a regression test); the expect() calls moved into readiness.rs remain backed by the existing TCP host/port validation in Adapter::new; and the zip example mirrors examples/fastapi-zip with AutoPublishAlias added so SnapStart applies to the alias the API integration targets. Prior findings about the README's contradictory "protected" claim, the 0/negative timeout semantics, the "before giving up" wording, and the publish = false decision are addressed in this revision.
The doc comments had grown to carry review history, measurements and repeated explanation. Cut roughly 200 lines net, keeping the part that stops a wrong edit and dropping the narrative -- the detail lives in git history where it belongs. src/snapstart.rs is down from 125 to 80 comment lines, src/lib.rs from 1200 to 1039. No behavior or test changes; the same tests pass.
| - Add SnapStart support. The adapter notifies your web application at the SnapStart | ||
| boundary via two opt-in HTTP hooks — `AWS_LWA_SNAPSTART_BEFORE_CHECKPOINT_PATH` | ||
| (before checkpoint) and `AWS_LWA_SNAPSTART_AFTER_RESTORE_PATH` (after restore) — | ||
| so it can drain and re-establish connections. Each hook call is bounded by a |
There was a problem hiding this comment.
[GENERAL] The changelog states the hook calls are time-bounded, but the implementation deliberately does not bound them:
so it can drain and re-establish connections. Each hook call is bounded by a
60-second timeout.
SnapStartHooks::post_hook has no timeout wrapper, and its own doc comment says the opposite:
/// Deliberately unbounded: Lambda bounds both phases already (the init budget, and
/// the function timeout for the after-restore hook). A fixed adapter-side cap was
/// unreachable for one phase and killed legitimate slow drains in the other.
I grepped src/snapstart.rs for any 60 / from_secs(6…) / hook-timeout constant and found none. README.md, docs/guide/src/features/snapstart.md, and both example READMEs all correctly say "The adapter does not impose its own deadline on a hook" — the changelog is the only place carrying the 60s claim, so it reads as a guarantee operators do not have. (The PR description repeats it too.) Suggest dropping that sentence and matching the wording used in the guide.
| | `AWS_LWA_SNAPSTART_BEFORE_CHECKPOINT_PATH` | Inner-app path the adapter POSTs to before a SnapStart snapshot | None | | ||
| | `AWS_LWA_SNAPSTART_AFTER_RESTORE_PATH` | Inner-app path the adapter POSTs to after a SnapStart restore | None | | ||
| | `AWS_LWA_POOL_IDLE_TIMEOUT_SECONDS` | Idle keep-alive (seconds) for the adapter's connection to your app | `4` | | ||
| | `AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS` | Seconds (fractional allowed, e.g. `0.5`) the adapter waits for the app to report ready (cold-start init **and** after a SnapStart restore). On expiry the adapter **fails** rather than serving: cold-start init fails (the runtime never starts) and a restore fails. Unset, `0`, or a negative value all mean **wait indefinitely** (no bound); a set-but-`<= 0` or malformed value is ignored with a `warn!`. The `async_init` path keeps its own ~9.8s bound (non-fatal) and is unaffected. | unset / `<= 0` (unbounded) | |
There was a problem hiding this comment.
[GENERAL] The scope documented for AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS omits the before-checkpoint phase, which is a new failure mode this PR introduces. The table says:
Seconds ... the adapter waits for the app to report ready (cold-start init and
after a SnapStart restore). On expiry the adapter fails rather than serving:
cold-start init fails (the runtime never starts) and a restore fails.
But before_snapshot also gates on readiness and propagates the timeout error, so an expiry there fails the checkpoint, not init or restore:
self.ensure_ready(&self.client, "before-checkpoint").await?;
Self::post_hook(&self.client, &self.domain, path).await?;ensure_ready uses the same readiness_timeout field, and the error string is "SnapStart before-checkpoint readiness check timed out after ...". The changelog's own bug-fix entry acknowledges this ("Both hooks now wait for the readiness check first, bounded by AWS_LWA_READINESS_CHECK_TIMEOUT_SECONDS"), so the reference table and the matching README.md row are inconsistent with both the code and the changelog. This matters most for AWS_LWA_ASYNC_INIT users, who are the reason the gate exists and who can now hit a checkpoint failure attributable to a variable documented as only affecting init and restore.
Everything else I checked from the prior review threads is resolved in this revision: the sync-init timeout now returns Err instead of unwrap_or_default(); readiness_check_timeout_from_env uses Duration::try_from_secs_f64 so 1e20 warns instead of panicking; hook_target canonicalizes the configured side through domain.set_path(configured) and checks root-collapse after canonicalization, so both sides of the guard share one normalization; canonicalize_hook_path now strips control bytes (blocking /snapstart/after%0A) and strips ; matrix params; the duplicated comment block is gone; the Adapter::new rustdoc and canonicalize_hook_path's fail-open contract match the code; base_client_pooling() restores the AWS_LAMBDA_INITIALIZATION_TYPE keying; and the guide now carries the same "guard only exists when the adapter is in the request path" warning as the example READMEs. Items you explicitly dismissed (container-image SnapStart, unreleased layer/image pins, publish = false) are excluded.
|
Superseded by #837, which is the same tree squashed into a single commit. Closing this one; the review history here stays readable and the |
Summary
Adds Lambda SnapStart support to the Lambda Web Adapter.
What it does
src/snapstart.rs: registers a SnapStart resource with the Lambda runtime andbridges the before-checkpoint / after-restore lifecycle to the inner web app over HTTP.
AWS_LWA_SNAPSTART_BEFORE_CHECKPOINT_PATHandAWS_LWA_SNAPSTART_AFTER_RESTORE_PATH.not reachable by external callers.
restored_client(write-onceOnceLock) so no pre-snapshot connections are reused.build_client()extraction,register_and_run()dedup of therun()arms,fetch_responsereturnsBoxBody<Bytes, Error>.Docs & examples
examples/fastapi-snapstart(container image) andexamples/fastapi-snapstart-zip(zip),both wiring the hook env vars through the SAM template.
Testing
cargo build— cleancargo test— 78 tests pass (4 e2e tests ignored, as they require deployed infrastructure)cargo clippy --all-targets— cleanNote:
nextestwas unavailable in this environment, so tests ran viacargo test -- --test-threads=1to preserve the env-var isolation the SnapStart config tests rely on.